Recorded media hotword trigger suppression

ABSTRACT

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for suppressing hotword triggers when detecting a hotword in recorded media are disclosed. In one aspect, a method includes the actions of receiving, by a computing device, audio corresponding to playback of an item of media content. The actions further include determining, by the computing device, that the audio includes an utterance of a predefined hotword and that the audio includes an audio watermark. The actions further include analyzing, by the computing device, the audio watermark. The actions further include based on analyzing the audio watermark, determining, by the computing device, whether to perform speech recognition on a portion of the audio following the predefined hotword.

CROSS-REFERENCE TO RELATED APPLICATION

This patent application is a continuation of, and claims priority under35 U.S.C. § 120 from, U.S. patent application Ser. No. 16/537,033, Aug.9, 2019, which is a continuation of U.S. patent application Ser. No.15/614,213, filed on Jun. 5, 2017. The disclosures of these priorapplications are considered part of the disclosure of this applicationand are hereby incorporated by reference in their entireties.

TECHNICAL FIELD

This specification generally relates to automated speech recognition.

BACKGROUND

The reality of a speech-enabled home or other environment—that is, onein which a user need only speak a query or command out loud and acomputer-based system will field and answer the query and/or cause thecommand to be performed—is upon us. A speech-enabled environment (e.g.,home, workplace, school, etc.) can be implemented using a network ofconnected microphone devices distributed throughout the various rooms orareas of the environment. Through such a network of microphones, a userhas the power to orally query the system from essentially anywhere inthe environment without the need to have a computer or other device infront of him/her or even nearby. For example, while cooking in thekitchen, a user might ask the system “how many milliliters in threecups?” and, in response, receive an answer from the system, e.g., in theform of synthesized voice output. Alternatively a user might ask thesystem questions such as “when does my nearest gas station close,” or,upon preparing to leave the house, “should I wear a coat today?”

Further, a user may ask a query of the system, and/or issue a command,that relates to the user's personal information For example, a usermight ask the system “when is my meeting with John!” or command thesystem “remind me to call John when I get back home.”

SUMMARY

For a speech-enabled system, the users' manner of interacting with thesystem is designed to be primarily, if not exclusively, by means ofvoice input. Consequently, the system, which potentially picks up allutterances made in the surrounding environment including those notdirected to the system, must have some way of discerning when any givenutterance is directed at the system as opposed, e.g., to being directedat an individual present in the environment. One way to accomplish thisis to use a hotword, which by agreement among the users in theenvironment, is reserved as a predetermined word that is spoken toinvoke the attention of the system. In an example environment, thehotword used to invoke the systems attention are the words “OKcomputer.” Consequently, each time the words “OK computer” are spoken,it is picked up by a microphone, conveyed to the system, which mayperform speech recognition techniques or use audio features and neuralnetworks to determine whether the hotword was spoken and, if so, awaitsan ensuing command or query. Accordingly, utterances directed at thesystem take the general form [HOTWORD] [QUERY], where “HOTWORD” in thisexample is “OK computer” and “QUERY” can be any question, command,declaration, or other request that can be speech recognized, parsed andacted on by the system, either alone or in conjunction with the servervia the network.

In addition to detecting a hotword, a computing device may be configuredto detect audio watermarks embedded in the audio of the hotword. Theaudio watermark may be a high frequency watermark that is inaudible tohumans or a watermark that sounds similar to background noise or static.The computing device may be configured to respond differently tohotwords depending on the audio watermark. For example, a business maycreate a commercial that includes an actor speaking. “Ok computer, giveme directions to Computer Store.” To prevent any computing device withinthe vicinity of a television playing the commercial from providingdirections to computer store, the creators of the commercial may includean audio watermark that overlaps the hotword. A computing devicedetecting the audio of the commercial may extract the audio watermarkand compare the audio watermark to an audio watermark database. Theaudio watermark database may include a rule for when a computing deviceshould perform speech recognition on the audio following the hotword andthis particular audio watermark and additional rules for other audiowatermarks.

The rule for the audio watermark for the commercial may includeconditions that a computing device should satisfy before furtherprocessing the audio from the commercial. An example rule may be that acomputing device should respond if the computing device is not currentlyproviding directions is not moving greater than ten miles per hour, is asmart speaker, and is located at the user's home. If the computingdevice satisfies these rules, then the computing device may performspeech recognition on the portion following the hotword. If thetranscription is, “give me directions to computer store,” then thecomputing device may display or audibly provide directions to computerstore. If the computing device does not satisfy the rules, then thecomputing devices does not perform speech recognition on the portion ofthe audio following the hotword and performs no further actions inresponse to the audio.

In some implementations, the audio watermark may encode data that mayeliminate the need for the computing device to compare the audiowatermark to an audio watermark database. The encoded data may include arule for when the computing device should perform speech recognition, anidentifier, an action, or any other similar data. In someimplementations, the computing device may use the encoded data incombination with the audio watermark database to determine whether toperform speech recognition on the audio following the hotword.

According to an innovative aspect of the subject matter described inthis application, a method for suppressing hotword triggers whendetecting a hotword in recorded media includes the actions of receiving,by a computing device, audio corresponding to playback of an item ofmedia content; determining, by the computing device, that the audioincludes an utterance of a predefined hotword and that the audioincludes an audio watermark; analyzing, by the computing device, theaudio watermark; and based on analyzing the audio watermark,determining, by the computing device, whether to perform speechrecognition on a portion of the audio following the predefined hotword.

These and other implementations can each optionally include one or moreof the following features. The action of analyzing the audio watermarkincludes comparing the audio watermark to one or more audio watermarks.The action of determining whether to perform speech recognition on theportion of the audio following the predefined hotword is based furtheron comparing the audio watermark to one or more audio watermarks. Theaudio watermark is an inaudible portion of the audio corresponding toplayback of an item of media content. The actions further include, basedon analyzing the audio watermark, identifying a source of the audiocorresponding to playback of the item of media content. The action ofdetermining whether to perform speech recognition on the portion of theaudio following the predefined hotword is further based on the source ofthe audio corresponding to playback of the item of media content. Theactions further include, based on analyzing the audio watermark,identifying a source of the audio corresponding to playback of an itemof media content; and updating a log file to indicate the source of theaudio corresponding to playback of the item of media content.

The audio watermark is included in a portion of the audio that includesthe utterance of the predefined hotword. the actions further includedetermining a type of the additional computing device. The action ofdetermining whether to perform speech recognition on the portion of theaudio following the predefined hotword is further based on the type ofthe additional computing device. The action of determining, by thecomputing device whether to perform speech recognition on a portion ofthe audio following the predefined hotword includes determining toperform speech recognition on the portion of the audio following thepredefined hotword. The actions further include generating, by anautomated speech recognizer, a transcription of the audio following thepredefined hotword, and performing an action corresponding to thetranscription of the audio following the predefined hotword. The actionof determining, by the computing device, whether to perform speechrecognition on a portion of the audio following the predefined hotwordincludes determining not to perform speech recognition on the portion ofthe audio following the predefined hotword. The actions further includesuppressing an action corresponding to the audio corresponding toplayback of the item of media content.

The actions further include determining a location of the additionalcomputing device. The action of determining whether to perform speechrecognition on the portion of the audio following the predefined hotwordis further based on the location of the additional computing device. Theactions further include determining a user setting of the additionalcomputing device. The action of determining whether to perform speechrecognition on the portion of the audio following the predefined hotwordis further based on the user setting of the additional computing deviceThe action of determining, by the computing device, that the audioincludes an utterance of a predefined hotword and that the audioincludes an audio watermark includes determining that the audio includesan utterance of a predefined hotword; and, based on determining that theaudio includes an utterance of a predefined hotword, determining thatthe audio includes the audio watermark. The action of determining, bythe computing device, that the audio includes an utterance of apredefined hotword and that the audio includes an audio watermarkincludes determining that the audio includes an utterance of apredefined hotword; and after determining that the audio includes anutterance of a predefined hotword, determining that the audio includesthe audio watermark.

The action of analyzing the audio watermark includes extracting datathat is encoded in the audio watermark. The action of determiningwhether to perform speech recognition on a portion of the audiofollowing the predefined hotword is further based on the data that isencoded in the audio watermark. The actions further include, based onanalyzing the audio watermark, identifying a type of media content ofthe audio corresponding to playback of an item of media content, andupdating a log file to indicate the type of media content of the audiocorresponding to playback of the item of media content. The actionsfurther include, based on analyzing the audio watermark, identifying atype of media content of the audio corresponding to playback of the itemof media content. The action of determining whether to perform speechrecognition on the portion of the audio following the predefined hotwordis further based on the type of media content of the audio correspondingto playback of the item of media content. The actions further include,based on analyzing the audio watermark, determining, by the computingdevice, whether to perform natural language processing on a portion ofthe audio following the predefined hotword.

Other embodiments of this aspect include corresponding systems,apparatus and computer programs recorded on computer storage deviceseach configured to perform the operations of the methods.

Particular embodiments of the subject matter described in thisspecification can be implemented so as to realize one or more of thefollowing advantages. A computing device may respond to hotwords thatinclude only specific audio watermarks, thus saving battery power andprocessing capacity of the computing device. Network bandwidth may bepreserved with fewer computing devices performing search queries uponreceiving hotwords with audio watermarks.

The details of one or more embodiments of the subject matter describedin this specification are set forth in the accompanying drawings and thedescription below. Other features aspects, and advantages of the subjectmatter will become apparent from the description, the drawings, and theclaims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates an example system for suppressing hotword triggerswhen detecting a hotword in recorded media.

FIG. 2 is a flowchart of an example process for suppressing hotwordtriggers when detecting a hotword in recorded media.

FIG 3 is an example of a computing device and a mobile computing device.

Like reference numbers and designations in the various drawings indicatelike elements.

DETAILED DESCRIPTION

FIG. 1 illustrates an example system 100 for suppressing hotwordtriggers when detecting a hotword in recorded media. Briefly, and asdescribed in more detail below, computing devices 102 and 104 receiveaudio 108 output from an audio source 106 (e.g., a television). Theaudio 108 includes an utterance of a predefined hotword and an audiowatermark. Both computing devices 102 and 104 process the audio 108, anddetermine that the audio 108 includes the predefined hotword. Thecomputing devices 102 and 104 identify the audio watermark. Based on theaudio watermark and the context or characteristics of the computingdevices 102 and 104, each of the computing devices 102 and 104 mayperform speech recognition on the audio.

In the example shown in FIG. 1, the audio source 106 is playing acommercial for Nugget World During the commercial, an actor in thecommercial says the utterance 108, “Ok computer, what's in a nugget?”The utterance 108 includes the hotword 110 “Ok computer” and a query 112“what's in a nugget?” The audio source 106 outputs the utterance 108through a speaker. Any computing device in the vicinity with amicrophone is able to detect the utterance 108.

The audio of the utterance 108 includes an audible portion 114 and anaudio watermark 116. The creator of the commercial may add the audiowatermark 116 to ensure particular computing devices responds correctlyto the utterance 108. In some implementations, the audio watermark 116may include audio frequencies that are higher or lower than humanhearing range. For example, the audio watermark 116 may includefrequencies that are greater than 20 kHz or less than 20 Hz. In someimplementations, the audio watermark 116 may include audio that iswithin the human hearing range but is not detectable by humans becauseof its sounds similar to noise. For example, the audio watermark 116 mayinclude a frequency pattern between 8 and 10 kHz. The strength ofdifferent frequency bands may be imperceptible to a human, but may bedetectable by a computing device. As illustrated by the frequency domainrepresentation 118 of the audio 108 includes an audio watermark 116 thatis in a higher frequency range than the audible portion 114.

The computing devices 102 and 104 may be any type of device that iscapable of receiving audio through a microphone. For example, computingdevices 102 and 104 can be a desktop computer, laptop computer a tabletcomputer, a wearable computer, a cellular phone, a smart phone, a musicplayer, an e-book reader, a navigation system, a smart speaker and homeassistant, wireless (e.g., Bluetooth) headset, hearing aid, smart watch,smart glasses, activity tracker, or any other appropriate computingdevice. As illustrated in FIG. 1, computing device 102 is a smart phoneand computing device 104 is a desktop computer. The audio source 106 canbe any source of audio such as, for example, a television, a radio, amusic player, a desktop computer, laptop computer, a tablet computer, awearable computer, a cellular phone, or a smart, phone. As illustratedin FIG. 1, the audio source 106 is a television.

The computing devices 102 and 104 each receive the audio through amicrophone. With respect to computing device 102, the microphone may bepart of an audio subsystem 120. The audio subsystem 120 may includebuffers filters, analog to digital converters that are each designed toinitially process the audio received through the microphone The buffermay store the current audio received through the microphone andprocessed by the audio subsystem 120. For example, the buffer stores theprevious five seconds of audio data. Similarly, the microphone ofcomputing device 104 may be part of an audio subsystem 122. The audiosubsystem 122 may include buffers, filters, analog to digital convertersthat are each designed to initially process the audio received throughthe microphone. The buffer may store the current audio received throughthe microphone and processed by the audio subsystem 122. For example,the buffer stores the previous three seconds of audio data.

The computing devices 102 and 104 each include a hotworder 124 and 126,respectively. The hotworders 124 and 126 are each configured to identifyhotwords in audio received through the microphone and/or stored in thebuffer. In some implementations, the hotworders 124 and 126 may beactive at any time that the computing devices 102 and 104 are poweredon. The hotworder 124 continuously analyzes the audio data stored in thebuffer. The hotworder 124 computes a hotword confidence score thatreflects the likelihood that current audio data in the buffer includes ahotword. To compute the hotword confidence score, the hotworder 124 mayextract audio features from the audio data such as filterbank energiesor mel-frequency cepstral coefficients. The hotworder 124 may useclassifying windows to process these audio features such as by using asupport vector machine or a neural network. In some implementations, thehotworder 124 does not perform speech recognition to determine a hotwordconfidence score. The hotworder 124 determines that the audio includes ahotword if the hotword confidence score satisfies a hotword confidencescore threshold. For example, the hotworder 124 determines that theaudio that corresponds to utterance 108 includes the hotword 110 if thehotword confidence score is 0.8 and the hotword confidence scorethreshold is 0.7. The hotworder 126 may functional similarly to thehotworder 124.

The computing devices 102 and 104 each include an audio watermarkidentifier 128 and 130 respectively. The audio watermark identifiers 128and 130 are each configured to process the audio received through themicrophone and/or stored in the buffer and identify audio watermarksthat are included in the audio. The audio watermark identifiers 128 and130 may each be configured to detect spread spectrum and psychoacousticshaping types of watermarks. These types of watermarks may be located infrequency bands that overlap the frequency bands of the correspondingaudio. A human may perceive these types of watermark as noise The audiowatermark identifiers 128 and 130 may also each be configured to detecthigh frequency watermarks. These types of watermarks may be located infrequency bands that are above the frequency bands of the correspondingaudio. The frequency bands of the high frequency watermarks may be abovethe threshold of human hearing. The audio watermark identifiers 128 and130 may also each be configured to detect low frequency watermarks.These types of watermarks may be located in frequency bands that arebelow the frequency bands of the corresponding audio. The frequencybands of the low frequency watermarks may be below the threshold ofhuman hearing. In some implementations, the audio watermark identifiers128 and 130 process the audio in response to the respective hotworder124 and 126 detecting a hotword.

The audio watermark identifiers 128 and 130 may each be configured toseparate the audio watermark and the primary audio. The primary audiomay be the portion of the audio to which the audio watermark was added.For example, the primary audio may include the audible portion 114 whichincludes the audio corresponding to “Ok computer, what's in a nugget?”without watermark 116. The audio watermark identifier 128 separates theaudio 118 that was received through the microphone of the computingdevice 102 into the primary audio 132 and the audio watermark 134.Similarly, the audio watermark identifier 130 separates the audio 118that was received through the microphone of the computing device 104into the primary audio 136 and the audio watermark 138. In someimplementations, audio watermark and the primary audio may overlap inthe time domain.

In some implementations, the audio watermark identifiers 128 and 130 mayprocess the audio watermarks 134 and 138, respectively, to identify acorresponding bitstream of the audio watermark. For example, the audiowatermark identifier 128 may process the audio watermark 134 anddetermine that the audio watermark 134 corresponds to a bitstream of0101101110101. The audio watermark identifier 130 may preform similarprocessing on the audio watermark 138.

The audio watermark comparers 140 and 144 each compare the respectiveaudio watermarks 134 and 138 to the audio watermarks 142 and 146respectively. For example, the audio watermark comparer 140 may comparethe frequency pattern or bitstream of watermark 134 to the audiowatermarks 142. The audio watermark comparer 140 may determine that theaudio watermark 134 matches an audio watermark of Nugget World. Theaudio watermark comparer 144 may make a similar determination.

The audio watermarks 142 and 146 may contain audio watermarks thatvarious entities embed in the audio of commercials or other distributedor broadcast audio that contain a hotword. Nugget World may include awatermark in the audio 108 to ensure what only particular devicesrespond to the hotword, perform speech recognition on the audio, andexecute the query 112. Nugget World may provide the audio watermark 116for inclusion in the audio watermarks 142 and 146 along withinstructions for when devices should respond to the hotword with theaudio watermark 116. For example, Nugget World may include in the audiowatermarks 142 and 146 instructions for any device with a Nugget Worlduser identifier that are located in a Nugget World restaurant to respondto the hotword with the audio watermark 116. In some implementations,the audio watermarks 142 and 146 are stored on the computing devices 102and 104 and updated periodically, for example, once per day. In someimplementations, the audio watermarks 142 or 146, the audio watermarkidentifiers 128 and 130, and/or the audio watermark comparers 140 and144 may be located on a remote server. In this instance, the computingdevice 102 or 104 may communicate with the remote server over a network.

The computing device 102 extracted the audio watermark 134 and matchesthe audio watermark 134 to a Nugget World watermark. Based on theinstructions in the audio watermarks 142 for the Nugget World watermark,the computing device 102 may perform speech recognition on the primaryaudio 132 and execute any queries or commands included in thecorresponding transcription. The instructions may include a set of rulesfor the computing device 102 to follow to determine whether to performspeech recognition.

The computing device 102 includes a location detector 156. The locationdetector 156 may generate geographic location data that reflects thelocation of the computing device. The location detector 156 may use anygeographic location technique, for example, GPS, triangulation, and/orany other similar locating technique. In some implementations, thelocation detector 156 may access map or location data that indicateslocations of various points of interest, such as businesses. Thelocation detector 156 may further identify a point of interest orbusiness where the computing device is located. For example, thelocation detector 156 may determine that the computing device 102 islocated at Nugget World.

The computing device 102 includes a device identifier 158. The deviceidentifier 158 includes a device identity 160 that identifies the typeof device for computing device 102. The device identity 160 may be adesktop computer, laptop computer, a tablet computer, a wearablecomputer, a cellular phone, a smart phone, a music player, an e-bookreader, a navigation system, a smart speaker and home assistant, or anyother appropriate computing device. For example, the device identity 160of computing device 102 is phone.

The computing device 102 includes a user identifier 162. The useridentifier 162 includes a user identity 164 that identifies the user forcomputing device 102. The user identity 164 may be an email address,phone number, or any other similar type of unique user identifier. Forexample, the user identity 164 of computing device 102 is user@examplecom. The user identifier 162 may be input by the user 154.

The computing device 102 includes user settings 152. The user settings152 may be provided by the user 154 and may include additional rules forhow the computing device 102 should respond to a hotword. For example,the user settings 152 may include a rule that the computing device 102does not respond to any hotwords that include audio watermarks unlessthe computing device 102 receives the hotword that includes an audiowatermark while at home. As another example, the user settings 152 mayinclude a rule that the computing device 102 does not responds tohotwords that include audio watermarks that correspond to particularbusinesses, such as Chicken World. In some implementations, the user 154may agree to allow the computing device 102 to respond to hotwords witha watermark of a particular business in exchange for a shopping discountor similar incentive.

In the example illustrated in FIG. 1, speech recognizer 166 remainsinactive as indicated by the speech recognizer status 168. The computingdevice 102 set the speech recognizer status 168 to inactive based onapplying of the instructions corresponding to the audio watermark 134 asstored in the audio watermarks to the device location, user settings152, device identity 160, and the user identity 164. For example, theinstructions corresponding to the audio watermark 134 may be to set thespeech recognizer status 168 to active if the user identity 164 is aNugget World identifier and the device is located at a Nugget Worldrestaurant. For computing device 102, the user identity 164 is not aNugget World identifier. Therefore, the speech recognizer status 168 isinactive.

In some implementations, the user interface generator 148 of thecomputing device 102 may provide, to a display of the computing device,data for a graphical interface. The graphical interface may indicate theprocesses or actions of the computing device 102 while, in advance of,or after the computing device has performed the processes or actions.For example, the user interface generator 148 may display an interfaceindicating that the computing device 102 is processing received audio,that the computing device 102 is identifying the audio watermark 134,the speech recognizer status 168, and/or any properties or rules of theidentified audio watermark 134.

In some implementations, the user interface generator 148 may generatean interface indicating that the speech recognizer status 168 isinactive. The interface may also include a user selectable option tooverride the speech recognizer status 168. For example, user 154 mayselect an option to set the speech recognizer status 168 to active.Having heard the query 112 “what's in a nugget,” the user 154 may becurious and request that the computing device 102 process the query 112and provide an output.

In some implementations, the computing device 102 may include an audiowatermark log 170. The audio watermark log 170 may include dataindicating a number of times that the computing device 102 has receivedeach audio watermark. For example, each time that the computing device102 receives and identifies audio watermark 134, the computing device102 may store data indicating the receipt of the audio watermark 134 inthe audio watermark log 170. The data may include a timestamp, a devicelocation, any relevant user settings user identifiers, and any othersimilar information. In some implementations, the computing device 102may provide the data of the audio watermark log 170 to an aggregateaudio watermark log on a server that combines the audio watermark logsfrom different computing devices that receive audio watermarks. Theaggregate audio watermark log may include the user identities for thereceiving computing device, a device identifier, and data stored In theaudio watermark log 170. In some implementations the data in theaggregate audio watermark log and the audio watermark log 170 maysynchronize. In this instance, the audio watermark log 170 may includeadditional log data from different devices as well as data identifyingthe different devices, different users, location information, timestampdata, and other related information.

In some implementations, the instructions for a particular audiowatermark may include instructions related to the data stored in theaudio watermark log 170. The instructions may relate to a particularnumber of times that a hotword marked with a particular audio watermarkshould activate s speech recognizer. For example, the instructions mayindicate that within 24-hour period, the audio watermark 116 should onlyactivate the speech recognizer 166 once.

In some implementations, the creator of the commercial on audio device106 may access the aggregate audio watermark log to identify the detailsrelated to each time the hotword 110 and corresponding audio watermark116 activated a speech recognizer. In some implementations, a user mayindicate through the user settings on the device for the computingdevice not to upload the audio watermark log to the aggregate audiowatermark log. In some implementations, a creator, or owner, of thecommercial on audio device 106 may pay a fee to the entity operating theaudio watermark system each time an audio watermark of the creator isprocessed by a computing device or each time an audio watermark of thecreator activates a speech recognizer of a computing device.

The computing device 104 processes the audio watermark 138 in a similarfashion to the computing device 102 processing the audio watermark 134.In particular, the computing device 104 extracted the audio watermark138 and matches the audio watermark 138 to a Nugget World watermark.Based on the instructions in the audio watermarks 146 for the NuggetWorld watermark, the computing device 102 may perform speech recognitionon the primary audio 136 and execute any queries or commands included inthe corresponding transcription. The instructions may include a set ofrules for the computing device 104 to follow to determine whether toperform speech recognition.

The computing device 104 includes a location detector 176. The locationdetector 176 may generate geographic location data that reflects thelocation of the computing device. The location detector 176 may use anygeographic location technique, for example, GPS, triangulation, and/orany other similar locating technique. In some implementations, thelocation detector 176 may access map or location data that indicateslocations of various points of interest, such as businesses. Thelocation detector 176 may further identify a point of interest orbusiness where the computing device 104 is located. For example, thelocation detector 176 may determine that the computing device 104 islocated at Nugget World.

The computing device 104 includes a device identifier 178. The deviceidentifier 178 includes a device identity 180 that identifies the typeof device for computing device 104. The device identity 180 may be adesktop computer, laptop computer, a tablet computer, a wearablecomputer, a cellular phone, a smart phone, a music player, an e-bookreader, a navigation system, a smart speaker and home assistant or anyother appropriate computing device. For example, the device identity 180of computing device 104 is desktop computer.

The computing device 104 includes a user identifier 182. The useridentifier 182 includes a user identity 184 that identifies the user forcomputing device 104. The user identity 184 may be an email address,phone number, or any other similar type of unique user identifier. Forexample, the user identity 184 of computing device 108 isstore@nuggetworld.com. The user identifier 182 may be input by a user.

The computing device 104 includes user settings 186. The user settings186 may be provided by a user and may include additional rules for howthe computing device 104 should respond to a hotword. For example, theuser settings 186 may include a rule that the computing device 104 doesnot respond to any hotwords that include audio watermarks unless thecomputing device 104 is located at a Nugget World restaurant. As anotherexample, the user settings 186 may include a rule that the computingdevice 104 does not respond to any hotwords other than those marked withaudio watermarks from Nugget World. As another example the user settings186 may indicate that the computing device 104 does not respond to anyhotwords with any type of audio watermark outside of Nugget World'sbusiness hours.

In the example illustrated in FIG. 1, speech recognizer 172 is active asindicated by the speech recognizer status 174. The computing device 104set the speech recognizer status 174 to active based on applying of theinstructions corresponding to the audio watermark 138 as stored in theaudio watermarks to the device location, user settings 186, deviceidentity 180, and the user identity 184. For example, the instructionscorresponding to the audio watermark 134 may be to set the speechrecognizer status 174 to active if the user identity 184 is a NuggetWorld identifier and the device is located at a Nugget World restaurant.For computing device 104, the user identity 184 is a Nugget Worldidentifier and the location is at a Nugget World. Therefore, the speechrecognizer status 174 is active.

The speech recognizer 172 performs speech recognition on the primaryaudio 136. The speech recognizer 172 generates the transcription “what'sin a nugget.” The computing device 104 may provide the transcription toa search engine if the transcription corresponds to a query. Thecomputing device may execute a command if the transcription correspondsto a command. In the example in FIG. 1, the computing device 104provides the transcription of the primary audio 136 to a search engine.The search engine returns a result, and the computing device 104 mayoutput the result through a speaker. For example, the computing device104 may output the audio 190 of “a nugget contains chicken.” In someimplementations, the user interface generator 150 may display the searchresults on the display of the computing device 104.

In some implementations, the user interface generator 150 may provideadditional interfaces. The graphical interface may indicate theprocesses or actions of the computing device 104 while in advance of, orafter the computing device 104 has performed the processes or actions.For example, the user interface generator 150 may display an interfaceindicating that the computing device 104 is processing received audio,that the computing device 104 is identifying the audio watermark 138,the speech recognizer status 174, and/or any properties or rules of theidentified audio watermark 138.

In some implementations, the user interface generator 150 may generatean interface indicating that the speech recognizer status 174 is active.The interface may also include a user selectable option to override thespeech recognizer status 174. For example, a user may select an optionto set the speech recognizer status 174 to suppress any actions relatedto the transcriptions. In some implementations, the user interfacegenerator 150 may generate an interface to update the user settings 186based on the recently received override and the current properties ofthe computing device 104. The user interface generator 148 may alsoprovide a similar interface after receiving an override command.

In some implementations, the computing device 104 may include an audiowatermark log 188. The audio watermark log 188 may store data similar toaudio watermark log 170 based on the audio watermarks received by thecomputing device 104. The audio watermark log 188 may interact with anaggregate audio watermark log in a similar fashion to the audiowatermark log 170.

In some implementations, the computing devices 102 and 104 may performspeech recognition on the primary audio 134 and 138 respectively,independent of the rules stored in the audio watermarks 142 and 146. Theaudio watermarks 142 and 146 may include rules related to performingactions of the primary audio based in part on the transcription.

FIG. 2 illustrates an example process 200 for suppressing hotwordtriggers when detecting a hotword in recorded media. In general, theprocess 200 performs speech recognition on audio corresponding to mediacontent based on the audio including a hotword and a watermark. Theprocess 200 will be described as being performed by a computer systemcomprising one or more computers, for example, the computing devices 102or 104 as shown in FIG. 1.

The system receives audio corresponding to playback of an item of mediacontent (210). In some implementations, the audio may be receivedthrough a microphone of the system. The audio may correspond to theaudio of a commercial played on a television or the radio.

The system determines that the audio includes an utterance of apredefined hotword and an audio watermark (220). In someimplementations, the audio watermark is an inaudible portion of theaudio. For example, the audio watermark may be located in frequencybands that are above or below those of human hearing. In someimplementations, the audio watermark is audible, but sounds similar tonoise. In some implementations, the audio watermark overlaps the audioof the predefined hotword In some implementations, the system determinesthat the audio includes the predefined hotword. In response to thatdetermination, the system processes the audio to determine whether theaudio includes an audio watermark.

The system compares the audio watermark to one or more audio watermarks(230). In some implementations, the system may compare the audiowatermark to a database of audio watermarks. The database may be storedon the system or on a different computing device. The system may comparea digital representation of the audio watermark or an analogrepresentation of the audio watermark in the time and/or frequencydomain. The system may identify a matching audio watermark and processthe audio according to a rule specified in the database for theidentified audio watermark. In some implementations, the system mayidentify a source, or owner, of the audio watermark. For example, thesource, or owner, may be the entity, Nugget World. The system may updatea log file to indicate that the system received a hotword with a NuggetWorld audio watermark.

The system based on comparing the audio watermark to the one or moreaudio watermarks, determines whether to perform speech recognition on aportion of the audio following the predefined hotword (240). Based onthe rule specified in the database for identified audio watermark, thesource of the audio watermark, and the context of the system, the systemdetermines whether to perform speech recognition on the audio followingthe predefined hotword. The context of the system may be based on anycombination of a type of the system, a location of the system, and anyuser settings. For example, the rule may specify that mobile phones thatare located at the user's home should perform speech recognition on theaudio upon receiving a hotword with a particular watermark from themanagement company of the apartment where the user lives. In someimplementations, the system determines, based on comparing the audiowatermark to the one or watermarks or based on analyzing the audiowatermark, whether to perform natural language processing on the portionof the audio following the predefined hotword The system may performnatural language processing in additional to the speech recognition orin place of the speech recognition.

In instances where the system determines to perform speech recognition,the system generates a transcription of the audio that follows thehotword. The system executes a command included in the transcriptionsuch as adding a calendar appointment for an apartment building meeting,or submits a query to a search engine. The system may output the searchresults over a speaker of the system or on the display of the system orboth.

In instances where the system determines not to perform speechrecognition, the system may remain in sleep mode standby mode, orlow-energy mode. The system may be in sleep mode standby mode, orlow-energy mode while processing the audio and may remain in sleep mode,standby mode, or low-energy mode if the system does not perform speechrecognition on the audio. In some implementations, the user 154 may beusing the computing device 102 while the computing device 102 receivesthe audio 118. For example, the user 154 may be listening to music orviewing a photo application. In this instance, the hotword and audiowatermark processing may occur in the background and the user'sactivities may be uninterrupted. In some implementations, the audio maynot include an audio watermark. In this instance the system may performspeech recognition on the audio after the hotword and execute anycommand or query included in the audio.

In some implementations, the system may determine a type of mediacontent for the audio. The system may compare the audio watermark to theaudio watermarks included in the audio watermark database. The systemmay identify a matching audio watermark in the audio watermark database,and the matching audio watermark may identify a type of media contentfor that particular audio watermark. The system may apply the rule forthe identified type of media content. For example, an audio watermarkdatabase may indicate that an audio watermark is included in sales mediadirectional media, commercial media, political media, or any other typeof media. In this instance, the system may follow a general rule for thetype of media. For example, the rule may be to only perform speechrecognition for commercial media when the system is located at home. Therule may also be a rule that is specific to the received audiowatermark. In some implementations, the system may also record the typeof media content in the audio watermark log.

In some implementations, the system may analyze the audio watermark. Thesystem may analyze the audio watermark in place of or in combinationwith comparing the audio watermark to the audio watermark database. Theaudio watermark may encode an action, an identifier, a rule, or anyother similar data The system may decode the audio watermark and processthe audio according to the decoded audio watermark. The audio watermarkmay be encoded as a header and payload. The system may identify theheader, which may be common to all or almost all audio watermarks or mayidentify particular groups of audio watermarks. A payload may follow theheader and encode the action, identifier, rule, or other similar data.

The system may apply the rule encoded in the audio watermark. Forexample, the rule may be for the system to perform speech recognition onthe portion of the audio following the hotword if the system is a smartspeaker that is located at place of business that corresponds to theuser identifier stored in the system. In this instance, the system mayhave no need to access an audio watermark database. In someimplementations, the system may add the rule encoded in the audiowatermark to an audio watermark database.

The system may use the data encoded in the audio watermark incombination with the audio watermark database. For example, data encodedin the audio watermark may indicate that the audio is political mediacontent. The system may access a rule that corresponds to the audiowatermark and that specifies for the system to perform speechrecognition on audio that includes the political media content watermarkor a commercial media content watermark when the system is a located inthe user's home. In this instance, the audio watermark may include aheader or other portion that the system may use to identifycorresponding audio watermarks in the audio watermark database. Thepayload may encode the type of media content or other data such asaction, identifier, or rule.

FIG. 3 shows an example of a commuting device 300 and a mobile computingdevice 350 that can be used to implement the techniques described here.The computing device 300 is intended to represent various forms ofdigital computers, such as laptops, desktops, workstations, personaldigital assistants, servers, blade servers, mainframes, and otherappropriate computers. The mobile computing device 350 is intended torepresent various forms of mobile devices, such as personal digitalassistants, cellular telephones, smart-phones, wireless (e.g.,Bluetooth) headsets, hearing aid, smart watches, smart glasses, activitytrackers, and other similar computing devices. The components shownhere, their connections and relationships, and their functions, aremeant to be examples only, and are not meant to be limiting.

The computing device 300 includes a processor 302, a memory 304, astorage device 306, a high-speed interface 308 connecting to the memory304 and multiple high-speed expansion ports 310, and a low-speedinterface 312 connecting to a low-speed expansion port 314 and thestorage device 306. Each of the processor 302 the memory 304, thestorage device 306, the high-speed interface 308, the high-speedexpansion ports 310, and the low-speed interface 312, are interconnectedusing various busses, and may be mounted on a common motherboard or inother manners as appropriate. The processor 302 can process instructionsfor execution within the computing device 300, including instructionsstored in the memory 304 or on the storage device 306 to displaygraphical information for a GUI on an external input/output device, suchas a display 316 coupled to the high-speed interface 308. In otherimplementations, multiple processors and/or multiple buses may be used,as appropriate, along with multiple memories and types of memory. Also,multiple computing devices may be connected, with each device providingportions of the necessary operations (e.g., as a server bank, a group ofblade servers, or a multi-processor system).

The memory 304 stores information within the computing device 300. Insome implementations, the memory 304 is a volatile memory unit or units.In some implementations, the memory 304 is a non-volatile memory unit orunits. The memory 304 may also be another form of computer-readablemedium, such as a magnetic or optical disk.

The storage device 306 is capable of providing mass storage for thecomputing device 300. In some implementations, the storage device 306may be or contain a computer-readable medium, such as a floppy diskdevice a hard disk device, an optical disk device, or a tape device, aflash memory or other similar solid state memory device, or an array ofdevices, including devices in a storage area network or otherconfigurations Instructions can be stored in an information carrier. Theinstructions, when executed by one or more processing devices (forexample, processor 302), perform one or more methods, such as thosedescribed above. The instructions can also be stored by one or morestorage devices such as computer- or machine-readable mediums (forexample, the memory 304, the storage device 306, or memory on theprocessor 302).

The high-speed interface 308 manages bandwidth-intensive operations forthe computing device 300, while the low-speed interface 312 manageslower bandwidth-intensive operations. Such allocation of functions is anexample only. In some implementations, the high-speed interface 308 iscoupled to the memory 304, the display 316 (e.g., through a graphicsprocessor or accelerator), and to the high-speed expansion ports 310,which may accept various expansion cards (not shown). In theimplementation, the low-speed interface 312 is coupled to the storagedevice 306 and the low-speed expansion port 314. The low-speed expansionport 314, which may include various communication ports (e.g., USB,Bluetooth, Ethernet, wireless Ethernet) may be coupled to one or moreinput/output devices, such as a keyboard, a pointing device, a scanner,a microphone, speakers, or a networking device such as a switch orrouter, e.g., through a network adapter.

The computing device 300 may be implemented in a number of differentforms, as shown in the figure. For example, it may be implemented as astandard server 320, or multiple times in a group of such servers. Inaddition, it may be implemented in a personal computer such as a laptopcomputer 322. It may also be implemented as part of a rack server system324. Alternatively, components from the computing device 300 may becombined with other components in a mobile device (not shown), such as amobile computing device 350. Each of such devices may contain one ormore of the computing device 300 and the mobile computing device 350,and an entire system may be made up of multiple computing devicescommunicating with each other.

The mobile computing device 350 includes a processor 352, a memory 364,an input/output device such as a touch-enabled display 354 acommunication interface 366, and a transceiver 368, among othercomponents. The mobile computing device 350 may also be provided with astorage device, such as a micro-drive or other device, to provideadditional storage. Each of the processor 352, the memory 364, thedisplay 354, the communication interface 366, and the transceiver 366,are interconnected using various buses, and several of the componentsmay be mounted on a common motherboard or in other manners asappropriate.

The processor 352 can execute instructions within the mobile computingdevice 350, including instructions stored in the memory 364. Theprocessor 352 may be implemented as a chipset of chips that includeseparate and multiple analog and digital processors. The processor 352may provide, for example, for coordination of the other components ofthe mobile computing device 350, such as control of user interfaces,applications run by the mobile computing device 350, and wirelesscommunication by the mobile computing device 350.

The processor 352 may communicate with a user through a controlinterface 358 and a display interface 356 coupled to the display 354.The display 354 may be, for example, a TFT (Thin-Film-Transistor LiquidCrystal Display) display or an OLED (Organic Light Emitting Diode)display, or other appropriate display technology. The display interface356 may comprise appropriate circuitry for driving the display 354 topresent graphical and other information to a user. The control interface358 may receive commands from a user and convert them for submission tothe processor 352. In addition, an external interface 362 may providecommunication with the processor 352, so as to enable near areacommunication of the mobile computing device 350 with other devices. Theexternal interface 362 may provide, for example, for wired communicationin some implementations, or for wireless communication in otherimplementations, and multiple interfaces may also be used.

The memory 364 stores information within the mobile computing device350. The memory 364 can be implemented as ore or more of acomputer-readable medium or media, a volatile memory unit or units, or anon-volatile memory unit or units. An expansion memory 374 may also beprovided and connected to the mobile computing device 350 through anexpansion interface 372, which may include, for example, a SIMM (SingleIn Line Memory Module) card interface. The expansion memory 374 mayprovide extra storage space for the mobile computing device 350, or mayalso store applications or other information for the mobile computingdevice 350. Specifically, the expansion memory 374 may includeinstructions to carry out or supplement the processes described above,and may include secure information also. Thus, for example, theexpansion memory 374 may be provided as a security module for the mobilecomputing device 350, and may be programmed with instructions thatpermit secure use of the mobile computing device 350. In addition,secure applications may be provided via the SIMM cards, along withadditional information, such as placing identifying information on theSIMM card in a non-hackable manner.

The memory may include, for example, flash memory and/or NVRAM memory(non-volatile random access memory), as discussed below. In someimplementations, instructions are stored in an information carrier thatthe instructions, when executed by one or more processing devices (forexample, processor 352), perform one or more methods, such as thosedescribed above. The instructions can also be stored by one or morestorage devices, such as one or more computer- or machine-readablemediums (for example, the memory 364, the expansion memory 374, ormemory on the processor 352). In some implementations, the instructionscan be received in a propagated signal, for example, over thetransceiver 368 or the external interface 362.

The mobile computing device 350 may communicate wirelessly through thecommunication interface 366, which may include digital signal processingcircuitry where necessary. The communication interface 366 may providefor communications under various modes or protocols, such as GSM voicecalls (Global System for Mobile communications), SMS (Short MessageService), EMS (Enhanced Messaging Service), or MMS messaging (MultimediaMessaging Service), CDMA (code division multiple access), TDMA (timedivision multiple access), PDC (Personal Digital Cellular), WCDMA(Wideband Code Division Multiple Access), CDMA2000, or GPRS (GeneralPacket Radio Service), among others. Such communication may occur, forexample, through the transceiver 368 using a radio-frequency. Inaddition, short-range communication may occur, such as using aBluetooth, WiFi, or other such transceiver (not shown). In addition, aGPS (Global Positioning System) receiver module 370 may provideadditional navigation- and location-related wireless data to the mobilecomputing device 350, which may be used as appropriate by applicationsrunning on the mobile computing device 360.

The mobile computing device 350 may also communicate audibly using anaudio codec 360, which may receive spoken information from a user andconvert it to usable digital information. The audio codec 360 maylikewise generate audible sound for a user, such as through a speaker,e.g., n a handset of the mobile computing device 350. Such sound mayinclude sound from voice telephone calls, may include recorded sound(e.g., voice messages, music files, etc.) and may also include soundgenerated by applications operating on the mobile computing device 350.

The mobile computing device 350 may be implemented in a number ofdifferent forms, as shown in the figure. For example, it may beimplemented as a cellular telephone 380. It may also be implemented aspart of a smart-phone 382, personal digital assistant, or other similarmobile device.

Various implementations of the systems and techniques described here canbe realized in digital electronic circuitry, integrated circuitry,specially designed ASICs (application specific integrated circuits),computer hardware firmware, software, and/or combinations thereof. Thesevarious implementations can include implementation in one or morecomputer programs that are executable and/or interpretable on aprogrammable system including at least one programmable processor, whichmay be special or general purpose, coupled to receive data andinstructions from, and to transmit data and instructions to, a storagesystem, at least one input device, and at least one output device.

These computer programs (also known as programs, software, softwareapplications or code) include machine instructions for a programmableprocessor, and can be implemented in a high-level procedural and/orobject-oriented programming language, and/or in assembly/machinelanguage. As used herein, the terms machine-readable medium andcomputer-readable medium refer to any computer program product,apparatus and/or device (e.g., magnetic discs, optical disks, memory,Programmable Logic Devices (PLDs)) used to provide machine instructionsand/or data to a programmable processor, including a machine-readablemedium that receives machine instructions as a machine-readable signal.The term machine-readable signal refers to any signal used to providemachine instructions and/or data to a programmable processor.

To provide for interaction with a user, the systems and techniquesdescribed here can be implemented on a computer having a display device(e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor)for displaying information to the user and a keyboard and a pointingdevice (e.g., a mouse or a trackball) by which the user can provideinput to the computer. Other kinds of devices can be used to provide forinteraction with a user as well; for example, feedback provided to theuser can be any form of sensory feedback (e.g., visual feedback,auditory feedback, or tactile feedback); and input from the user can bereceived in any form, including acoustic, speech, or tactile input.

The systems and techniques described here can be implemented in acomputing system that includes a back end component (e.g., as a dataserver), or that includes a middleware component (e.g. an applicationserver), or that includes a front end component (e.g., a client computerhaving a graphical user interface or a Web browser through which a usercan interact with an implementation of the systems and techniquesdescribed here), or any combination of such back end, middleware orfront end components. The components of the system can be interconnectedby any form or medium of digital data communication (e.g., acommunication network). Examples of communication networks include alocal area network (LAN), a wide area network (WAN), and the Internet.

The computing system can include clients and servers. A client andserver are generally remote from each other and typically interactthrough a communication network. The relationship of client and serverarises by virtue of computer programs running on the respectivecomputers and having a client-server relationship to each other.

Although a few implementations have been described in detail above,other modifications are possible. For example, white a clientapplication is described as accessing the delegate(s), in otherimplementations the delegate(s) may be employed by other applicationsimplemented by one or more processors, such as an application executingon one or more servers. In addition, the logic flows depicted in thefigures do not require the particular order shown, or sequential order,to achieve desirable results. In addition, other actions may beprovided, or actions may be eliminated, from the described flows, andother components may be added to, or removed from, the describedsystems. Accordingly, other implementations are within the scope of thefollowing claims.

What is claimed is:
 1. A computer-implemented method when executed ondata processing hardware of a user device causes the data processinghardware to perform operations comprising: receiving audio data capturedby a microphone of the user device while the user device is in a sleepmode, the audio data corresponding to playback of an item of mediacontent output from an audio source in a vicinity of the user device;processing, while the user device remains in the sleep mode, the audiodata to determine a corresponding bitstream of an audio watermarkencoded in the audio data; and based on the determined correspond ngbitstream of the audio watermark encoded in the audio data, determining,without accessing an audio watermark database to identify a matchingaudio watermark, to bypass performing speech recognition on the audiodata.
 2. The computer-implemented method of claim 1, wherein the audiodata comprises an utterance of a command preceded by a hotword. 3 . Thecomputer-implemented method of claim 2, wherein determining to bypassperforming speech recognition on the audio data comprises bypassingperforming speech recognition on a portion of the received audio datacorresponding to the utterance of the command. 4 . Thecomputer-implemented method of claim 2, wherein the audio watermark isencoded in a portion of the audio data that includes the hotword.
 5. Thecomputer-implemented method of claim 2, wherein the operations furthercomprise, prior to processing the audio data to determine thecorresponding bitstream of the audio watermark encoded in the audiodata, determining that the received audio data includes the hotword. 6.The computer-implemented method of claim 5, wherein determining that thereceived audio data includes the hotword comprises: computing, withoutperforming speech recognition, a hotword confidence score indicating alikelihood that the audio data includes the hotword; and determiningthat the hotword confidence score satisfies a hotword confidence scorethreshold.
 7. The computer-implemented method of claim 1, whereinprocessing the audio data while the user device remains in the sleepstate further comprises processing the audio data to identify the audiowatermark encoded in the audio data by detecting a spread spectrumshaping type of watermark.
 8. The computer-implemented method of claim1, wherein the user device remains in the sleep mode after determiningto bypass performing speech recognition on the audio data.
 9. Thecomputer-implemented method of claim 1, wherein the operations furthercomprise: analyzing the audio watermark to identify the audio source ofthe audio data corresponding to playback of the item of media content,wherein determining to bypass performing speech recognition on the audiodata is further based on the audio source of the audio datacorresponding to playback of the item of media content.
 10. Thecomputer-implemented method of claim 9, wherein the operations furthercomprise updating a log file to indicate the source of the audio datacorresponding to playback of the item of media content.
 11. A userdevice comprising: data processing hardware, and memory hardware incommunication with the data processing hardware and storinginstructions, that when executed on the data processing hardware, causethe data processing hardware to perform operations comprising: receivingaudio data captured by a microphone of the user device while the userdevice is in a steep mode, the audio data corresponding to playback ofan item of media content output from an audio source in a vicinity ofthe user device; processing while the user device remains in the sleepmode the audio data to determine a corresponding bitstream of an audiowatermark encoded in the audio data; and based on the determinedcorresponding bitstream of the audio watermark encoded in the audiodata, determining, without accessing an audio watermark database toidentify a matching audio watermark, to bypass performing speechrecognition on the audio data.
 12. The user device of claim 11, whereinthe audio data comprises an utterance of a command preceded by ahotword. 13 The user device of claim 12, wherein determining to bypassperforming speech recognition on the audio data comprises bypassingperforming speech recognition on a portion of the received audio datacorresponding to the utterance of the command.
 14. The user device ofclaim 12, wherein the audio watermark is encoded in a portion of theaudio data that includes the hotword.
 15. The user device of claim 12,wherein the operations further comprise, prior to processing the audiodata to determine the corresponding bitstream of the audio watermarkencoded in the audio data, determining that the received audio dataincludes the hotword.
 16. The user device of claim 16, whereindetermining that the received audio data includes the hotword comprises.computing, without performing speech recognition, a hotword confidencescore indicating a likelihood that the audio data includes the hotword;and determining that the hotword confidence score satisfies a hotwordconfidence score threshold.
 17. The user device of claim 11, whereinprocessing the audio data while the user device remains in the sleepstate further comprises processing the audio data to identify the audiowatermark encoded in the audio data by detecting a spread spectrumshaping type of watermark.
 18. The user device of claim 11, wherein theuser device remains in the sleep mode after determining to bypassperforming speech recognition on the audio data
 19. The user device ofclaim 11, wherein the operations further comprise: analyzing the audiowatermark to identify the audio source of the audio data correspondingto playback of the item of media content, wherein determining to bypassperforming speech recognition on the audio data is further based on theaudio source of the audio data corresponding to playback of the item ofmedia content.
 20. The user device of claim 19, wherein the operationsfurther comprise updating a log file to indicate the audio source of theaudio data corresponding to playback of the item of media content.